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Introduction 


The ultimate goal of teacher evaluation systems is to 
improve the quality of instruction by clarifying 
expectations for effective teaching and helping 
teachers meet those expectations through high- 
quality feedback and support. Classroom 
observations — which make up the majority of a 
teacher's summative rating in most States and 
districts — give teachers the opportunity to receive 
meaningful and direct feedback about their practice. 
They can also inform the development of resources 
to help teachers address areas for improvement. But 
many observers still struggle to accurately assess 
teacher performance and give teachers high-quality 
feedback and tools to help them improve their 
instruction. 


States face several common challenges that prevent 
observers from providing more meaningful feedback 
to teachers. First, many States use observation 
rubrics that are cumbersome in length and/or lack 
specificity. Observers struggle to assess teacher 
performance if they are unclear on expectations for 
teacher practice or are trying to assess too many 
indicators at once. Second, most observers do not 
have the ability to give grade level- or content- 
specific feedback to all of the teachers in their 
caseload. This diminishes the specificity of the 
feedback that teachers receive about how well their 
instruction is aligned with college- and career-ready 
standards in their content area. Finally, in States and 
districts where principals are the primary observers, 
they often lack the time and skill to conduct rigorous 


observations and deliver high-quality feedback to all 
of their teachers. Furthermore, observer training that 
is infrequent or does not include opportunities to 
practice feedback falls short of what observers need 
in order to develop their observation skills. 


The good news is that strategies exist to address 
each of these challenges. States play an important 
role in improving the quality of observations and can 
employ a variety of mechanisms to make 
improvements, depending on their policy context. 
For example, States that have the authority to revise 
their observation frameworks can clarify vague 
content and eliminate redundancies or indicators 
that are not related to student outcomes. To support 
district observer training, States can create resources 
that guide observers in how to conduct observations 
and post-observation debriefs with teachers. Finally, 
States can seed innovation at the local level and 
elevate lessons and best practices from districts that 
develop their own observation tools, resources and 
policies. 


This publication summarizes research, lessons and 
resources gathered from States, districts and 
supporting organizations about how States can 
make observations more meaningful for teachers 
and observers. The appendix contains guiding 
questions for States about the effectiveness of their 
observation frameworks, examples of State plans to 
communicate with and solicit feedback from 
stakeholders, and an overview of the different 
approaches that States might take to improve their 
observation frameworks and tools. 


The Reform Support Network, sponsored by the U.S. Department of Education, supports the Race to 
the Top grantees as they implement reforms in education policy and practice, learn from each other, 
and build their capacity to sustain these reforms, while sharing these promising practices and lessons 
learned with other States attempting to implement similarly bold education reform initiatives. 


Overview of RSN 
Observation Frameworks 


In September 2014, the Reform Support Network (RSN) 
brought together four States—Delaware, Louisiana, 
North Carolina and Ohio—that were in various stages 
of evaluation system implementation and expressed 
interest in improving the classroom observation 
component of their evaluation systems. The purpose 
of the project was to help this cohort assess challenges 
with their current observation frameworks; identify 
goals to address those challenges; and develop tools, 
resources and processes to help them achieve their 
goals. Each State team came to the project with 

a specific focus on how it wanted to improve its 
observation system, based on their own observations 
or anecdotal evidence from the field: 


Leaders from North Carolina's Department of 
Public Instruction felt that their observation rubric — 
based on the Danielson Framework — overwhelmed 
observers with too many indicators. State leaders 
wanted to build a companion observation tool 

for the evaluation framework that would define 

the specific elements that observers should target 
when conducting classroom observations. 


In Delaware, most districts use a single observation 
model developed by the State. Leaders from the 
Delaware Department of Education sought to seed 
innovation by encouraging districts to develop, and 
submit to the State for approval, alternative models 
that reflect best practices in observation rubrics. 


+ In Louisiana, teachers and observers wanted 
greater specificity about instructional expectations 
for teachers of different grade levels and subject 
areas. In response, leaders from the Louisiana 
Department of Education sought to create 
content-specific instructional guides that provide 
details about what observers should look for in 
mathematics and English Language Arts classes for 
different grade bands. 


* Ohio Department of Education leaders noted 
that observers sometimes find it challenging to 
differentiate among performance levels and give 
high-quality feedback to teachers. They plan to 


implement a co-observation model, where two 
observers observe the same teacher and compare 
ratings, in order to improve rater accuracy and 
enhance the quality of observer feedback. 


Over the course of seven months, State leaders 
participated in webinars, an in-person convening, and 
one-on-one coaching from experts to help them meet 
these goals. Below are details about the strategies 
they chose to implement in order to improve their 
observation systems, as well as lessons learned by 
these States which may help other States that want 

to improve the quality of observations and support to 
teachers. 


Strategies to Improve 
Observation Systems 


The strategy a State chooses to improve the quality 

of classroom observations will depend on the specific 
outcome it hopes to achieve. It will also depend on the 
specific State context, including the State's relationship 
with districts and its resources, including time, money 
and staff. The strategies below reflect the diversity of 
goals and policy environments among the States in 
this cohort and can be adapted by other States to fit 
their particular context. 


Create clear, streamlined rubrics 


Research on classroom observations indicates that 
observation rubrics are most effective when: 


* Coherent: They are aligned with State teaching 
standards. 


* Concise: They are brief, condensed and easy for 
observers to use. 


* Clear: They use precise language to describe 
teacher and student behavior. 


* Focused: Indicators are directly related to student 
outcomes. (See Appendix A, “Questions to Identify 
High-Quality Observation Instruments.) 


'Measures of Effective Teaching (MET) Project, Foundations of 
Observation (2013); TNTP, Rating a Teacher Observation Tool (2011); 
and TNTP, Fixing Classroom Observations (2013). 


Reform Support Network 


Delaware leaders received feedback from teachers 
and leaders about several ways in which its statewide 
evaluation rubric does not meet the best practices 
described above. Some of the challenges include a 
lack of clarity that makes it difficult to gather evidence 
for some indicators, it may not be appropriate for 

all environments (for example, kindergarten), and it 
could be more rigorous and directly related to student 
outcomes. The State also received feedback from the 
field that the rubric does not support high-quality 
feedback to teachers. In response to this feedback, 

it invited districts to submit alternative evaluation 
models that incorporate the best practices described 
above and better meet educators’ needs. 


Develop supplemental tools that help 
observers focus on the right areas for 
development 


In some cases, States and districts will not have 

the flexibility to make significant changes to their 
observation rubrics without approval from State 
boards of education or State legislatures. This was 
true for North Carolina, where the evaluation system 
is based on the Danielson Framework. The team 
opted for a shorter-term solution: to develop a 
supplemental tool that observers could use when 
conducting classroom observations. This observation 
tool will contain a subset of the Danielson Framework 
indicators that are observable and aligned with 
student outcomes. 


In an effort to support principals, Louisiana State 
leaders developed a guidebook for principals that 

will serve as the hub of instructional leadership best 
practices and resources. Of particular importance 

was the focus on providing meaningful feedback to 
teachers. Louisiana adapted the instructional practice 
guides from Student Achievement Partners to create 
their own content-specific guides based on Louisiana's 
State content standards.’ These tools are meant to 
supplement the observation rubric that observers 
already use and to help guide post-observation 
debrief conversations with teachers. Additionally, the 


? Measures of Effective Teaching (MET) Project, Foundations of 
Observation (2013); TNTP, Rating a Teacher Observation Tool (2011); 
and TNTP. Fixing Classroom Observations (2013). 


Reform Support Network 


tools established a basis for the teacher support case 
studies located in the Louisiana Principals’ Teaching 
& Learning Guidebook. 


Establish meaningful structures that 
encourage strong, effective feedback 


Ohio is focused on ensuring that teachers receive 
meaningful feedback following observations. 

Because evaluators sometimes find it challenging 

to differentiate among performance levels on the 
evaluation rubric, they may need additional support 
to accurately assess teacher performance. They 

also theorized that if observers are not accurately 
assessing teacher performance, teachers are also 
likely not receiving high-quality feedback from the 
observers. Based on Tennessee's success with sending 
State coaches to co-observe with principals who 
struggled to assign accurate ratings and give high- 
quality feedback to teachers, Ohio plans to provide 
funding to a small group of districts to implement 
this strategy. Other benefits of co-observation include 
the opportunity for teachers to get content-specific 
feedback from peer teachers and for teacher leaders to 
develop their instructional leadership skills. 


Tennessee also requires that observers demonstrate 
that they can deliver high-quality feedback before 
they are certified to observe teachers. Following 
observer training, candidates in Tennessee must pass a 
certification exam developed by the National Institute 
for Excellence in Teaching, which assesses the quality 
of their post-observation feedback. Delaware has 
instituted a similar observer certification exam. 


Insights for Other States 


In addition to these strategies, States learned 
important lessons about their role in the continuous 
improvement of statewide initiatives and how to work 
with districts and educators to achieve common goals. 
As the Race to the Top grants conclude, these insights 
will be particularly important as States consider how 
to sustain the work they have done and continue to 
refine the systems they have put in place. 
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Solicit feedback from the field on State- 
developed tools and strategies 


To ensure that State-developed tools and processes 
are actually meeting the needs of educators, States 
should create opportunities for stakeholders to provide 
feedback to inform the development and ongoing 
refinement of these resources. Louisiana asked 
educators to provide feedback on the content of its 
draft instructional guides. It then made revisions and 
will disseminate to principals broadly in summer 2015 
as a resource included in the Louisiana Principals’ 
Teaching & Learning Guidebook. Mindful of the 
value of feedback from the field, Louisiana leaders will 
work to strategically collect feedback on how both 
principals and teachers use the guides in their practice 
to help inform future iterations of the tools, as well 

as training on how to use them (see Appendix C). In 
2015-2016, direct support initiatives for principals 

will take place across the State, including a 16-month 
principal fellowship. This initiative will bring together 
more than 100 principals in the first year with a focus 
on instructional leadership, including the use of these 
newly developed tools. 


Pilot new strategies with districts 


Districts are critical partners in the implementation 

of teacher evaluation systems. They have stronger 
connections to teachers and principals than State 
education agencies (SEAs), are able to closely monitor 
implementation and facilitate observer trainings, and 
often have more flexibility than the State to refine 
observation tools and systems. To the extent that 
States allow and even encourage it, districts can serve 
as centers of innovation to try new observation 
strategies; produce tools that benefit teachers and 
observers; and, in some cases, even create new 
observation frameworks. 


After receiving substantial feedback from educators 
that the statewide evaluation model does not always 
meet their needs, Delaware developed a process 

by which local educational agencies (LEAs) could 
create their own educator evaluation systems and 
processes “in the spirit of increasing educator support, 


accountability, and student achievement’? Districts 
then apply to the State for approval to implement 
these alternative models. The benefit of this strategy to 
the districts is clear: They can implement an evaluation 
model that better meets the needs of their teachers 
and observers. The benefit to the State is that it can 
collect evidence from multiple evaluation systems and 
continue to refine its statewide model to reflect best 
practices. 


Create a communications strategy that 
incentivizes districts and builds trust 


Delaware has long offered districts the opportunity to 
innovate and create their own observation rubrics if 
they determined that the statewide observation rubric 
was not meeting their needs. But to the surprise of 
State leaders, no district ever submitted an alternative 
model (although a few charter schools have done 

so). As they conducted outreach to district leaders, 
they discovered that district leaders assumed that the 
bar for approval would be too high and that it would 
not be worth the effort. There has also been some 
acknowledgment that the State system has been 
built over the past decade and that it is not easy to 
build one’s own. Delaware is currently in the process 
of revising its Communications to districts about the 
opportunity in order to encourage more districts to 
take advantage of it. 


In Ohio, where State leaders want to encourage 
districts to use co-observation as a strategy to improve 
the quality of observations, they are developing an 
application process for districts so that they can apply 
to participate in a future co-observation pilot. While 
developing this application, they recognized the 
importance of communicating the benefits of co- 
observation to stakeholders (district leaders, teachers, 
principals, and these groups’ professional associations) 
to inform them about the opportunity and inspire 
them to participate (see Appendix B). 


3 http://www.doe.k1 2.de.us/cms/lib09/DE01922744/Centricity/ 
Domain/186/AlternativeEducatorEvalSystemApp_2015_FINAL. 
pdf 


Produce and disseminate tools that 
support educators and improve 


implementation 


One of the best ways for States to support districts 
and educators is by gathering feedback from the 
field about implementation challenges and then 
developing tools, resources and guidance to help 
address those challenges. 


North Carolina and Louisiana both used feedback 
from educators to determine that they needed to 
create tools that would improve the implementation 
of observations. North Carolina found through its 
review of observation data, as well as feedback from 
principals, that they were struggling to apply the 
evaluation framework to classroom observations 
because it contained many more indicators than what 
a principal typically observes. Therefore, the State team 
determined that principals needed a shorter version 
of the framework that contained only the observable 
indicators, which would make observations more 
efficient to conduct and targeted feedback easier to 
give. 


Similarly, Louisiana heard from teachers and principals 
that they wanted to receive and provide feedback 
specific to their grade and content area, and that all 
observers may not have the expertise to provide this 
feedback to teachers. In response, Louisiana State 
leaders created the content-specific guides described 
above. 


Conclusion 


The State efforts described above highlight the 
importance for States to continually monitor their 
evaluation systems to ensure that they are working as 
intended and are providing the right information to 
teachers about their practice. Data that indicate that 
systems are not working as they should can empower 
States to make improvements, whether through 
policy changes, the creation of tools and resources, 
professional development, or the dissemination of 
best practices. 


Delaware, Louisiana, North Carolina, and Ohio each 
took a slightly different approach to improving the 
quality of observations in their State, demonstrating 
the diverse roles that States can play to support 
educators as they implement observations. These roles 
include engaging stakeholders in the development 
of tools and resources, collecting feedback through 
outreach and data analysis, creating tools that address 
implementation challenges, and communicating 
effectively with stakeholders (for example, district 
leaders and educators) about the purpose of State- 
developed tools and strategies. 


5 


Appendix A - 
Questions to Identify High-Quality Observation Instruments 


High-quality observation instruments are the foundation of high-quality observation systems. The following 
are sample questions States might ask to determine whether their observation rubric meets a high standard of 
quality. 


1. Does the instrument measure observable behaviors of teaching practice? 
If not, modify to include only observable behaviors. 


Observable indicator Non-observable indicator 


“Teacher checks for student understanding and responds to “Teacher uses and aligns resources strategically.’ 
misunderstandings.’ 


2. Are the behaviors observable within a typical classroom observation? 
They must be observable to be considered for use in observations. 


Typically observed in one class Not typically observed in one class 


“Teacher presents content clearly.” “Teacher modifies instruction in response to data” 
p y 


3. Are the criteria student-centered, requiring evaluators to look for evidence of student engagement and 
learning? 
If not, revise performance expectations to show the desired result instead of teacher actions. 


Student-centered Teacher-centered 

“All or almost all hands are raised when reviewing taught “Teacher routinely emphasizes completion of work and 

material. More than half of hands are up on new material’ consistently encourages students to expend their best 
effort.’ 


4. Do the criteria for judgment require support materials (for example, lesson plans, student work)? 


If so, these criteria would be better assessed through a different evaluation component. 


Does not require support materials Requires support materials 
“Teacher implements routines to maximize instructional “Teacher develops standards-based units and long-term 
time” plans’ 


5. Do the criteria and tools set a high bar for performance, or do they settle for minimally acceptable 
performance? 
If not, modify to reflect high expectations for teacher performance. 


High expectations Compliance-driven 


“Teacher works to ensure that the needs of every student are “Teacher integrates a variety of technology tools 
met, providing extra support or enrichment as necessary.” and applications into instructional design and 
implementation” 


6. Are the performance expectations for teachers clear and precise? 
If not, add specificity so that observers can collect and teachers can reflect on concrete evidence. 


Clear and precise Unclear and imprecise 


“Teacher presents all information in a clear, well-organized, “Teacher displays extensive content knowledge of all the 
factually accurate manner without mistakes that would leave subjects that he or she teaches.’ 
students with misunderstandings at the end of a lesson? 


7. Are the criteria and tools concise enough for teachers and evaluators to understand thoroughly and use 
easily? 
If not, revise performance expectations to address only observable behaviors. 


Concise Not concise 
Observation framework consists of 5-6 performance Framework consists of 6+ indicators and/or is longer than 
indicators that fit on a single page. a single page. 


8. Does the instrument clearly differentiate among performance levels? 
If not, revise the language so that teachers and observers know what it takes to get to the next level. 


Developing Proficient 

“Teacher uses a variety of instructional strategies and “Teacher consistently and effectively uses a variety of 
questioning techniques to develop students’ higher- instructional strategies and questioning techniques to 
level thinking skills, although some strategies may not be develop students’ higher-level thinking skills.’ 
effective’ 


9. Does the instrument assess lesson content in addition to how the lesson is being taught? 
If not, revise the indicators to include specific guidance on how to assess content. 


Adequate Not adequate 


“The text(s) is at or above the complexity level expected for “Teacher states an objective that conveys what students 
the grade and time in the school year’ are learning and doing ina lesson. 


Examples of High-Quality Observation Rubrics 
TNTP Core Teaching Rubric: http://tntp.org/publications/view/tntp-core-teaching-rubric-a-tool-for- 
conducting-classroom-observations 


Student Achievement Partners Instructional Practice Guides: http://achievethecore.org/dashboard/410/sear 
ch/3/1/0/1/2/3/4/5/6/7/8/9/10/11/12/page/969/instructional-practice-guide-list-pg 


Insight Core Framework: http://www.insighteducationgroup.com/insight-core-framework-landing-page 


Examples of High-Quality Observation Rubrics 
TNTP, Fixing Classroom Observations (2013): http://tntp.org/assets/documents/TNTP_ 
FixingClassroomObservations_2013.pdf 


TNTP, Rating a Teacher Observation Tool (2011): http://tntp.org/assets/documents/TNTP_ 
RatingATeacherObservationTool_Feb2011.pdf 


MET Project, Foundations of Observation (2013): http://www.metproject.org/downloads/MET-ETS__ 
Foundations_of_Observation.pdf 


Appendix B 


The Ohio Department of Education, in conjunction with the RSN, developed the following plan for future use 

to communicate with stakeholders about the opportunity to pilot co-observations. The plan identifies each 
stakeholder group, as well as high-level messages and specific text to use with each group. It also includes a work 
plan with discrete tasks, a timeline, and owners assigned to each task. 


Draft Ohio Co-Observation Communications Plan 


Reference Documents 


RSN Framework for Communication and Engagement 
RSN Social Media Tip Sheets 
RSN Memo: Using Co-Observation to Impact Teacher Practice 


RSN Memo: Summary of Findings: Co-Observations in Teacher Evaluation 


Top Messages 
Message 
Teachers Are you interested in opportunities for teacher leadership? Are you a master teacher in your district 
hoping to influence other teachers? 
Principals Are you confident in your ability to provide useful feedback to teachers from classroom observations? 
ls that feedback leading to improvement? 
Districts What will it take for teacher evaluation to be a transformative tool for improving practice in your district? 
Associations Can you help the Ohio Department of Education connect your members with opportunities for 


leadership and resources? 


All Co-observation is a strategy that is validated by research and the experience of other States. 

All Co-observation develops teacher leaders and school leaders into stronger instructional coaches. 

All Co-observation yields more useable feedback for teachers as they work to improve their practice. 

Principals This is an opportunity for job-embedded, continual support to improve your observation practice in 
your school. It is not external, add-on training. 

All The Ohio Department of Education is making resources available to districts that want to use co- 
observation to increase the benefits of classroom observation for improving instructional practice. 

Districts with Master Co-observation is an opportunity to build collaborative leadership with Master Teachers and Endorsed 

Teachers/Teacher Teacher Leaders in your district to facilitate professional dialogue and collegial learning with other 


Leader Endorsement teachers. 


foe) 


Text for Communicating 


Associations 


Can you help the Ohio Department of Education connect your members with opportunities for leadership and 
resources? 


The Ohio Department of Education is making resources, including funding and training, available to districts 
that want to use co-observation to increase the benefits of classroom observation for improving instructional 
practice. 


Co-observation is a strategy that is validated by research and the experience of other States. During co- 
observation, a teacher is observed and gets feedback from the school leader and a teacher leader. Teachers 
receive more comprehensive feedback, and instructional leaders develop confidence and coaching skills. 


The co-observation process is a potential way to engage Master Teachers and Endorsed Teacher Leaders in 
providing useful feedback to other teachers to improve instruction. 


District Leaders — No Master Teachers/Endorsed Teacher Leaders 


What would it take for teacher evaluation to be a transformative tool for improving practice in your district? 


Co-observation is a strategy that is validated by research and the experience of other States. During co- 
observation, a teacher is observed and gets feedback from the school leader and a teacher leader. Teachers 
receive more comprehensive feedback, and instructional leaders develop confidence and coaching skills. 


The Ohio Department of Education is making resources available to districts that want to use co-observation to 
increase the benefits of classroom observation for improving instructional practice. 


District Leaders — Have Master Teachers/Endorsed Teacher Leaders 


What would it take for teacher evaluation to be a transformative tool for improving practice in your district? 


The co-observation grant is an opportunity to build collaborative leadership with Master Teachers and Endorsed 
Teacher Leaders in your district to facilitate professional dialogue and collegial learning with other teachers. 


Co-observation is a strategy that is validated by research and the experience of other States. During co- 
observation, a teacher is observed and gets feedback from the school leader and a teacher leader. Teachers 
receive more comprehensive feedback, and instructional leaders develop confidence and coaching skills. 


The Ohio Department of Education is making resources available to districts that want to use co-observation to 
increase the benefits of classroom observation for improving instructional practice. 


School Leaders 


Are you confident in your ability to provide useful feedback to teachers from classroom observations? Is that 
feedback leading to improvement? 


Co-observation is a strategy that is validated by research and the experience of other States. During co- 
observation, a teacher is observed and gets feedback from the school leader and a teacher leader. Teachers 
receive more comprehensive feedback, the school leader gains confidence as an observer, and teacher leaders 
develop as instructional coaches. 


The Ohio Department of Education is making resources available to districts that want to use co-observation to 
increase the benefits of classroom observation for improving instructional practice. 


Teachers 
+ Are you interested in opportunities for teacher leadership? 


The Ohio Department of Education is making resources available to districts that want to use co-observation to 
increase the benefits of classroom observation for improving instructional practice. 


During co-observation, a teacher is observed and gets feedback from the school leader and a teacher leader. 
The process improves the feedback teachers get on their instruction and helps the observers develop as 
instructional leaders. 


Ohio Communications Plan — Work Plan 


Communication Effort | Objectives I NUTeltvare=1 (3) Person Responsible Tentative Timeline 


nformal discussion with Lay the groundwork. Superintendents ODE Staff Summer 


pilot population 

Communicate to the pilot. Encourage superintendents Superintendents ODE Staff Late summer/early 

districts via email O apply. fall 

Personal outreach to Encourage superintendents Superintendents ODE Staff Phone outreach +1 

district superintendents to apply. week, +2 weeks, +3 

weeks 

Send targeted email nspire teachers, principals Teachers, ODE Director Timed with pilot 

to associations for and district staff to learn principals and population outreach 

forwarding more and encourage district staff 
superintendents to apply. 

State organization Inspire teachers, principals Teachers, ODE Director Timed with pilot 

conferences and district staff to learn principals and population outreach 

Webpage Inspire teachers, principals Teachers, ODE Staff Timed with pilot 
and district staff to learn principals and population outreach 
more and encourage district staff 
superintendents to apply. 

Teacher toolkit Inspire teachers, principals Teachers, ODE Staff One month before 
and district staff to learn principals and pilot population 
more and encourage district staff outreach 
superintendents to apply. 

Superintendents’ Encourage superintendents Superintendents ODE Staff One month before 

newsletter to apply. pilot population 

outreach target, 


Monday after pilot 
population outreach 


more and encourage district staff 
| superintendents to apply. 


Arts. States can adapt these questions to generate feedback from stakeholders about State-developed tools or 


Appendix C 
In partnership with the Louisiana Department of Education, the RSN developed the following plan to solicit 
feedback from educators about the State-developed instructional guides for Mathematics and English Language 
strategies. 


Instructional Guide Feedback Loops Plan Summary 


The RSN is providing individualized assistance to four States that want to improve their teacher observation 
processes and tools. Louisiana has created instructional guides to support the use of classroom observation to 
improve practice in the classroom so that observers can identify effective teaching practices and provide useful 
feedback. The RSN is supporting the development of a plan for gathering feedback on the guides from the field. 


Following are approaches for how Louisiana can build its feedback plan, including a list of potential survey 
questions and a plan for facilitating focus groups. 


Determine Research Objectives 


Louisiana has already identified several research questions: 
- Whois using the guides? 
+ Why do certain groups use it more than others? 


How are they using it? At what point during the observations is it used? What's the experience like? How does 
it translate to feedback? 


+ Did the tool result in a change in practice? For whom? 


It may be helpful to generate additional research questions. 


Gather Evidence Via Survey 

The first phase of gathering evidence may begin with an online questionnaire. Surveys diverge from the truth 
because of four sources of error: 

+ Sampling — if the sample is too small or not representative of the population 

+ Coverage - if some members of the population have a zero-percent chance of being sampled 

- Measurement — if a respondent's answer is inaccurate or imprecise 


« Non-Response - if respondents differ from non-respondents 


Louisiana should consider each source of error in determining a methodology for the survey. The survey sampling 
plan depends on the available frame, which is likely to be a list of email addresses. How complete is the email list? 
This will determine the extent of potential coverage error. Does the list include any other information about the 
address, such as the school where the person is employed or his/her role? The sampling plan should also consider 
the level of granularity with which the department would like to make estimates. For example, if estimates are 
desired at the district level, the sampling plan will need to accommodate an appropriate sample size within each 
district. 


; 


Because there is no cost to send additional online questionnaires, the department might consider sending the 
questionnaire to the entire list. This eliminates sampling error, but it does not eliminate the other sources of error, 
and it may contribute to “survey fatigue” in the field. 


lt is preferable to send a unique link to each email address. This can be done via survey software such as 
SurveyMonkey or Qualtrics. This prevents one person from submitting multiple responses, helps ensure 
respondents are the original recipients, and enables reminder emails to be sent only to those who have not yet 
responded. 


Remember, there is no magic response rate at which we can be sure of our estimates. What matters is whether 
and how the non-respondents differ from the respondents. This can be difficult to estimate, and the department 
should consider how to prioritize the management of non-response error. One option is to compare the non- 
respondents to the respondents on other known parameters (such as additional information contained in the 
original frame). Another is to seek out and interview a sample of non-respondents. 


Thoughtful questionnaire design can help mitigate measurement error. Questionnaire designers should also 
consider the likely concerns of respondents regarding confidentiality and potentially undesirable answers. Other 
strategies for improving the quality of questionnaires include: 


Review by content experts 
Cognitive interviews with respondents 
- A field test of limited scope before the primary collection 


RSN Deliverables 


The RSN will provide a bank of draft questionnaire iterns, guided by best practices in questionnaire design, as a 
resource for survey development (see Appendix C.1). 


Gather Evidence Via Focus Group 


The department may wish to use the survey results to invite participants to a focus group. In this case, it will 

be extremely important to provide accurate confidentiality information to respondents before they respond 
to the survey. A focus group can be used to delve into remaining research questions and to gather stories of 
implementation. 


RSN Deliverables 


The RSN will deliver a preliminary focus group guide geared toward the research objectives (see Appendix C.2). 


Use Findings to Inform Adjustments to Guides and Outreach 


The most important step will be to use the evidence gathered from the field to inform action steps for the 
design and use of the guides. The department should plan for the resources required to analyze and synthesize 
the findings of the survey and focus groups for decision makers, and create a facilitated meeting to determine 
appropriate actions. 
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Appendix C.1: Draft Questionnaire Items 


Best practices in survey methodology suggest that all questionnaires should begin with an introduction that 
includes the following: 
+ Adescription of the purpose of the survey and why responses are important 
An accurate estimate of the time needed to complete the questionnaire 
Accurate information about confidentiality and the use of responses 
+ The name and contact information of a person to contact with regard to any questions 


- Aclosing message thanking the respondent for his/her time 
Demographics 


If you do not have information associated with email addresses in your survey frame, you may wish to collect 
information through the questionnaire. Consider that some respondents may be reluctant to provide such 
information if they think that their responses may be used in a manner that will reflect negatively on them. You 
can make these items optional, but you will complicate your ability to sort the responses. These questions do not 
need to be presented at the beginning of the survey. 


Depending on the answers to these questions, you may use skip logic to avoid unnecessary questions. For 
example, if a respondent has not used the guides, they should not be asked to answer questions about the use of 
the guides. 


1. Please select your roles during this school year. [Provide a list of roles.] 


2. During the past school year, have you conducted any classroom observations of teachers? (Yes/No/Don't 
Know) 


3. [If yes to question 2] About how many classroom observations do you conduct in atypical month? [Provide a 
dropdown menu of single numbers with a reasonable range, such as 1-101] 


4. Please select your school district this school year. [Provide a dropdown menu of school districts.] 


Questions about Instructional Guides 


5. Are you aware of the Instructional Guides to Observation & Feedback provided by the State Department of 
Education? [Consider including a screenshot of the guide with this question.] (Yes/No/Don't Know) 


6. Have you reviewed the instructional guides? (Yes/No/Don't Know) 


wu 


7. [if yes to question 6] Did you use the instructional guides in any of the following ways? Select “yes;"“no,’ or “don't 
know" for each option: 


a. To gather evidence during a full lesson observation. 
b. To gather evidence during a focused classroom observation. 
c. To help prepare written or oral feedback to teachers. 


d. To inform my own understanding of instructional practices. 


e. To find resources related to standards-based instruction. 
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8. Are there any other ways in which you used the instructional guides this year? Please describe. [Provide text 
box.] 


9. Did you find the instructional guides helpful for [insert options from question 7, according to skip logic, for 
respondents who indicated that they had used the item in each way]? Please describe how the guides were 
helpful or not helpful for this purpose. [Provide text box.] 


10.Has your practice or the practice of teachers in your school changed as a result of these guides? (Yes/No/Don't 
Know) [If yes] Please describe the changes in practice. [Provide text box.] 


11.Would you be willing to participate in a focus group about the instructional guides? [If possible, include 
information about timing and location] 
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Appendix C.2: Draft Focus Group Guide 


A focus group is a useful tool for gathering evidence related to research questions. You can use focus groups to 
accomplish the following: 


Explore the range of possible answers. 
Find out what you don't know you don't know. 
Gather information from the participants’ interactions with each other. 


Record specific anecdotes. 


Focus Group Principles 


Aim for your focus group to include 6-8 people. Consider beginning with role-alike groups (for example, 
principals) to avoid confusion among participants and to remove barriers to candid feedback. Make sure that 
your feedback group participants represent the range of school characteristics in the State (for example, urban 
and rural, high-need). 


The moderator should aim to put participants at ease and remain neutral. 


The moderator should ask probing questions and balance letting the group explore the topic with staying 
focused on the main questions. 


The moderator should not be responsible for taking notes. If you can, it is helpful to record the session. 
Whatever method you decide to use, inform the participants of how their input will be captured. 


A typical focus group lasts 1.5-2 hours. 


Snacks are always a good idea. 
Process 


Opening/Ice Breaker 


Introduce yourself and the purpose of the focus group. 
Provide information about confidentiality, including how the State will use notes/recording from the meeting. 
Establish ground rules, such as: 
One person talks at a time. 
Be aware of your air time so that everyone has an opportunity to speak. 
Provide evidence and examples whenever possible. 
Speak clearly so everyone in the group can hear. 
Do not criticize other members’ opinions, but feel free to ask follow-up questions. 


Open the discussion by having each participant state his/her name and position. 
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Building Rapport 


> Start with a low-anxiety question. 
- Consider a round-robin for the first question — have everyone answer in turn. This ensures that everyone has 
a chance to speak and helps the moderator get a sense of the room. For example, “Can you provide a brief 


example of how you have used the instructional guide in your work?” 
In-Depth Investigation 


Interaction among the group becomes more prominent. 
Consider the following research questions: 

Who is using the guides? 

Why do certain groups use it more than others? 


How are they using it? At what point in observations is it used? What's the experience like? How does it 
translate to feedbac 


ro) 


Did the tool result in a change in practice? For whom? 


be sure to ask. 


+ If the participants don't volunteer information about their use of the instructional guides in specific situations, 
- Encourage discussion but keep the pace moving to ensure you cover priority areas. 


lf you begin to run out of time, provide an option for participants to give additional feedback, either by email or 
on a Post-It note. 


Closure 


+ Generally summarize what you heard without recounting everything point by point. 
» Repeat what actions will be taken as a result of their feedback, and how this will be communicated. 


Thank them for their participation. 
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Appendix D 


This chart describes the project goals and objectives for each State that participated in the cohort, as well as key 
lessons that each State learned about effective observation systems (content) and how State leaders can best 
cultivate these systems (process). 


Goal 


DE . 
LA . 
NC : 


o1 £ 


(0) 0) {Yeh dAY=) 


Make Support DDOE as they 
observations work with districts 
more to develop and 
meaningful submit for approval 
forobservers alternative evaluation 
and models in a way that 
educators. seeds innovation and 
develops stronger 
models. 
Improve Refine instructional 
quality of guides and develop a 
feedbackon “use case”to share as a 
content. model with LEAs prior 
to start of 2015-2016 
school year. 
Make Develop and pilot 


observations a companion tool, 

more efficient which is made up 

and targeted. of indicators that 
are observable and 
correlated with 
positive student 
outcomes, for the 
current evaluation 
framework. 


Increase district use 
of co-observation 
as a Strategy to help 


Improve 
accuracy of 
observation 


ratings observers increase the 
Improve quality of feedback for 
quality of teachers and rating 
feedback to Calibration among 
teachers. observers. 


Key Lessons — Content 


High-quality observation 
rubrics are clear, concise, 
coherent and focused. 


Teachers need 
meaningful feedback (for 
example, content-specific, 
actionable). 


Observers are not always 
confident when providing 


content-specific feedback. 


Annotating rubrics to 
give specific guidance 

on content can build 
confidence among 
observers as they provide 
this feedback. 


Rubrics are too 
burdensome for 
observers. 


They need more clarity, 
conciseness, connections 
to student outcomes. 


Co-observation can be an 
effective way to improve 
inter-rater reliability and 
quality of feedback. 


Co-observation also gives 
teachers a leadership 
opportunity. 


Key Lessons — Process 


Provide districts with the ability to 
propose alternative models and 
innovate within the system. 


Need to create a communications 
strategy that incentivizes districts 
and builds trust. 


Role of SEA is to produce and 
disseminate tools 


Important to ensure quality and 
usability in the field 


This is accomplished by soliciting 
stakeholder feedback throughout 
the development and upon 
implementation in the field. 


Pilot new strategy with small set of 
districts. 


Create a shared definition of high- 
quality instruction and feedback, as 
well as shared outcomes for student 
achievement. 


State needs a strong theory of 
action. 


Pilot a new strategy with a small set 
of districts. 


Use requests for proposals (RFPs) to 
encourage district innovation. 
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